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Abstract 

We  describe  the  various  techniques  that  have  been  proposed  for  constructing 
non-parametric  confidence  intervals  using  the  bootstrap.  These  include  bootstrap  pivotal  in¬ 
tervals,  percentile  and  bias-corrected  percentile  intervals,  and  non-parametric  tilting  intervals. 
These  methods  are  small  sample  improvements  over  the  usual  +t&  intervals.  We  discuss  them 
in  detail,  outlining  the  underlying  assumptions  in  each  case.  We  show  how  the  non-parametric 
tilting  interval  can  be  viewed  as  an  extension  of  a  bootstrap  pivotal  interval,  and  suggest  a 
number  of  generalizations.  Finally,  the  various  intervals  are  compared  in  a  small  simulation 
study. 
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1.  Introduction. 

Recently,  a  number  of  techniques  have  been  proposed  for  constructing  confidence  inter¬ 
vals  using  the  bootstrap  (see  Efron(1981)  and  Schenker(1983)).  These  techniques  are  non- 
parametric  in  nature,  and  are  designed  to  work  well  over  a  wide  variety  of  situations.  Because 
they  are  based  on  the  bootstrap,  they  can  be  used  in  situations  in  which  the  “parameter”  is  an 
extremely  complex  functional  of  the  distribution  and  an  exact  analysis  would  be'  impossible. 

In  this  paper,  we  describe  the  various  bootstrap  methods  that  have  been  suggested  for 
constructing  confidence  intervals  and  compare  them  in  a  few  examples.  The  paper  is  largely 
expository,  although  some  new  ideas  are  presented  for  the  construction  of  intervals  through 
non-parametric  tilting  (Sections  7  and  8).  Unfortunately,  we  offer  little  in  the  way  of  practical 
advice  for  choosing  among  the  various  intervals.  Research  in  this  area  is  greatly  needed. 


2.  The  Problem  and  Some  Notation. 

We  observe  zx,...zn  assumed  to  be  realizations  of  random  variables  Xi,...X„  ~  i.i.d  F. 
The  distribution  F  is  unknown  and  the  problem  is  to  construct  a  confidence  interval  for  the 
parameter  8  =  6(F).  By  a  confidence  interval,  we  mean  lower  and  upper  points  L  =  L(x i,  ...z„) 
and  U  =  U(xi,  ...zn)  such  that  P(L  <  8  <  U )  =  1  —  2a,  where  P(-)  denotes  probability  under 
the  true  distribution  F.  Since  the  intervals  are  to  be  non-parametric,  we  would  ideally  require 
that  this  hold  for  all  F.  None  of  the  intervals  described  here  claim  to  satisfy  this,  although  the 
non-parametric  tilting  interval  seems  to  come  closest.  We  will  confine  our  discussion  to  central 
intervals,  i.e.  intervals  (L,U)  such  that  P(8  <  L)  =  P(8  >U)  =  a.  Non-central  intervals  can 
be  obtained  through  obvious  modification. 

Given  Xi,X2,  ...Xn,  ( X%  can  be  a  scalar  or  vector  random  variable),  we  estimate  0  by 
$  =  8(F*)  where  F*is  the  empirical  distribution  function  of  Xi,...Xn.  The  observed  value  of 
8  is  60i,  =  6(Fn)  where  F„  is  the  empirical  distribution  function  of  xl,  ...x„. 

We  let  W  be  a  random  vector  with  Wi  >  0,  W,-  =  1  and  w  be  a  realization  of 

W .  Let  Fn(w)  be  the  distribution  putting  mass  to,-  on  z,-,  i  =  1,2 ,...n.  Many  of  the  tech¬ 
niques  will  utilize  “bootstrap  sampling” —  that  is,  sampling  from  zx,z2,...x„  with  replace¬ 
ment.  This  is  equivalent  to  sampling  W  from  the  rescaled  multinomial  Mult(n,w°)/n,  where 
«°  =  (1/n,  l,n,  ...1/n).  We’ll  use  ~  to  indicate  bootstrap  sampling  and  a  bootstrap  value 
obtained  in  this  way  will  be  denoted  by  0*  =  0(F„(»)).  We’ll  refer  to  a  bootstrap  sample 
either  by  its  weight  vector  w,  or  by  X*  =  (X{,X%,  ...X*).  Finally,  B  will  denote  the  empirical 
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distribution  function  of  8*  under  & 


(“the  bootstrap  distribution”). 


3.  Overview. 

FVequentist  confidence  intervals  are  based  on  a  test  function,  say  appropriate  for 

testing  H  :  0  =  8\ .  The  interval  is  constructed  as  follows.  For  each  trial  value  $i ,  we  include 
$i  in  our  confidence  interval  if  we  would  accept  H  in  a  1  —  2a  size  test  based  on  t(X ,  ^).  This 
procedure  requires  knowledge  of  the  distribution  of  t(X ,  0*)  for  each  9\.  Usually,  a  simplifying 
assumption  is  made —  that  t(X,$i)  is  pivotal ,  that  is,  has  a  distribution  not  depending  on 
$i.  With  this  assumption,  it  is  not  necessary  to  consider  each  trial  value  9\  separately.  We 
assume  some  parametric  distribution  for  t(X,  $i),  then  invert  the  pivotal  to  yield  the  confidence 
interval.  A  simple  example  is  Xx,  X2,  ...Xn  ~  1).  Then  a  confidence  interval  for  0  is  found 

by  inverting  the  pivotal  X  —  0 ,  whose  distribution  is  A/(0, 1  /n). 

The  Bootstrap  Pivotal,  Percentile  and  Bias-Corrected  Percentile  intervals  (Sections  4,  5, 
and  6)  are  non-parametric  analogues  of  parametric  pivotal  intervals.  The  pivotal  distribution 
is  not  assumed  known;  instead  it  is  estimated  non-parametrically  using  the  bootstrap.  In 
Sections  4  and  5  we  provide  the  “recipes”  for  constructing  these  intervals  and  outline  the 
underlying  assumptions.  In  Section  6,  we  discuss  the  appropriateness  of  the  various  intervals 
in  a  few  simple  problems. 

The  Non-Parametric  Tilting  interval  (Section  7)  is  more  ambitious  than  the  pivotal- 
based  techniques.  Instead  of  assuming  the  existence  of  a  pivotal,  it  attempts  to  estimate  the 
distribution  of  6  —  $i  for  each  trial  value  $i.  A  confidence  interval  is  then  formed  consisting  of 
values  of  $i  for  which  we  would  accept  the  hypthesis  H  :  0  =  $i. 

The  Estimated  Pivotal  interval  (Section  8)  is  a  compromise  between  the  two  approaches. 
The  data  is  used  to  suggest  an  appropriate  pivotal  quantity,  then  a  bootstrap  pivotal  interval 
is  constructed. 

In  Section  9  we  compare  all  the  intervals  in  two  numerical  examples. 
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4.  Bootstrap  Pivotal  Intervals. 


4.1.  The  Simple  Pivotal 

We  assume  that  0  —  0  is  a  pivotal  quantity,  that  is 

9-9~H  (j41) 

where  H  is  a  distribution  not  involving  0,  and  also  that  approximately 

9*  -  0ob.  H  (A2) 

Assumption  (A2)  is  based  on  the  premise  that  if  Fn  is  close  to  F,  the  bootstrap  distribution 
of  0*  —  0obt  will  be  close  to  that  of  0  —  0,  as  long  as  0{-)  is  a  reasonably  smooth  functional. 
Of  course,  if  If  is  a  continuous  distribution,  then  (A2)  is  at  best  an  approximation,  since  the 
bootstrap  distribution  is  necessarily  discrete.  The  intervals  described  in  this  section  and  the 
next  section  all  use  this  kind  of  bootstrap  approximation.  To  simplify  the  notation,  we  will 
ignore  the  fact  that  it  it  only  an  approximation. 

Under  (Al)  and  (A2),  we  have  1  —  2a  =  P(H~1(a)  <  0  —  9  <  H~1(  1  —  a))  =  P(0  — 
H-1(a)  <9  <9-  H~l{  1  -  a)). 

Substituting  0 for  0  and  noting  that  ff-1()  =  B-1(-)  —  Ooti,  we  obtain  the  Bootstrap 
Pivotal  interval: 

0  €  {26 ob,  -  B-*(l  -  a),20oht  -  B~l(c ,))  (1) 


4.2.  Other  Pivotals 

The  bootstrap  pivotal  interval  can  be  based  on  an  arbitrary  pivotal  t(X,  0),  as  long  as 
it  is  monotone  in  9.  We  assume  t(X,0)  ~  H,  t(X*,0obl)  ~  H ,  where  t(X,  0)  is  monotone 
decreasing  in  0.  Inverting  the  pivot  as  above,  we  obtain 

(2) 

where  t% 1(-)=inverse  of  £(•,  •)  with  respect  to  the  second  argument. 

The  bootstrap  pivotal  interval  is  used  by  Efron  (1981)  in  the  form  of  a  “bootstrap  t*  and 
by  Schenker(1983),  who  calls  it  the  “substitution  method*.  We  have  introduced  the  obvious 
name  “bootstrap  pivotal  interval*  here. 


4  Section  5:  Percentile  Intervals 

4.3.  The  Role  of  Nuisance  Parameters 

We  can  think  of  an  arbitrary  distribution  G  as  consisting  of  two  parts,  say  G  =  (0,if), 
where  0  =  0(G)  is  the  parameter  of  interest  and  if  =  if(G)  is  a  vector  of  nuisance  parameters, 
possibly  infinite  dimensional.  The  true  distribution  can  be  written  as  F  =  (0 <rue, if true).  With 
this  decomposition,  we  can  say  more  clearly  the  meaning  of  the  statement  at(X ,  0)  H,  H  not 
involving  0*.  What  we’re  really  assuming  is  that  F  is  a  member  of  some  family  of  distributions 
7  existing  in  the  space  of  possible  distributions.  The  members  of  7  correspond  to  different  0 
values  and  are  characterized  by  the  property  t(X,  0)  ~  H.  Because  of  this  pivotal  assumption, 
we  don’t  have  to  know  the  structure  of  (or  estimate)  the  entire  family  7.  Only  a  single  member 
of  7  need  be  estimated.  The  empirical  distribution  function  F„  estimates  that  member  (i.e. 
(Fn  =  (0obti  ifobt))j  and  from  this  we  obtain  the  distribution  H.  By  construction,  the  interval 
will  have  correct  coverage  for  F  G  7. 

A  family  like  7  also  underlies  the  percentile  and  bias-corrected  percentile  intervals  (dis¬ 
cussed  next).  The  non-parametric  tilting  interval  (Section  7)  doesn’t  make  a  pivotal  assump¬ 
tion,  and  essentially  tries  to  estimate  the  entire  family. 


5.  Percentile  Intervals. 


5.1.  Uncorrected  Intervals 


Here  we  assume  A1  and  A2,  and  further  that 

H  is  symmetric  around  0 

(A3) 

In  this  case,  the  pivotal  interval  (1)  becomes: 

0e(B-1(a),B-1(l-a)) 

(3) 

Efron  calls  this  the  Percentile  Interval  since  it  uses  the  percentiles  of  0* 

as  “percentiles*  of  0. 

5.2.  Generalization  of  the  Percentile  Interval 

If  a  symmetric  pivotal  exists  on  some  other  scale,  i.e. 

g(0)  -  g(0)  ~  H 

(A4) 

and 

g(h  -g{6eh,)~H 

(A5) 

and 
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with  H  symmetric  around  0  and  g(-)  is  an  unknown,  monotone  increasing  function,  then  as  in 
(3)  we  get  as  an  interval  for  g(0): 

s(»)e(e-1(«),c-1(i-°))  W 

where  G  is  the  distribution  function  of  g[0*).  Transforming  back  to  the  0  scale  gives 

*e(ir,(G-‘(‘'),sr,(<j-,U-«))  (5) 

or 

~  (6) 

which  is  again  the  percentile  interval .  Thus  the  percentile  interval  has  the  correct  coverage  if 
a  symmetric  pivotal  exists  on  any  scale.  Conveniently,  we  don’t  have  to  know  $(•)  because  the 
resultant  interval  doesn’t  depend  on  j(*). 

There  is  a  simple  connection  between  the  bootstrap  pivotal  interval  based  on  $  —  $  and 
the  percentile  interval.  Writing  {280bi  “  <B“X(1  —  a),2$0b9  “  B“x(a))  as  (0^9  -  [j?“l(l  -  a)  - 
Oobi}) ,  $0b9  +  [0ob9  —  B”1(a)]),  we  see  that  the  percentile  interval  is  the  bootstrap  pivotal  interval 
reflected  about  the  point 


5.3.  Bias-Corrected  Percentile  Intervals 


5.3.1  Normal  Correction, 

If  the  distribution  H  in  A4  and  A5  is  symmetric  around  u  ^  0,  the  percentile  interval 
will  be  biassed  and  will  not  have  the  correct  coverage.  This  would  occur  as  a  result  of  bias  in 
the  estimator  6,  It  turns  out  that  if  we  are  willing  to  assume  a  parametric  form  for  17,  then  u 
can  be  estimated  and  a  corrected  interval  can  be  derived.  As  was  the  case  for  the  percentile 
interval,  the  corrected  interval  will  not  depend  on  the  transformation  j(-). 

Since  P{g{6*)  <  =  <  §ob$),  we  can  use  the  latter  to  estimate  the  bias.  Using 

this  correction,  we  then  match  the  distributions  of  $(<?)  -  $(0)  and  g(6 *)  -  on  the  $(•) 

scale,  then  transform  back  to  the  $  scale. 

As  an  example,  suppose  we  choose  H  =  «A/(ti,  1).  Then 

g(0)-g(d)~M(  0,l)-«  (7) 


and 


~  90ob$)  ~  <V(0,1)  +  u 


(8) 
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We  can  solve  for  u  by  noting  that  P(g(0*)  <  g(60h))  =  $(-«)  =  G(g(60t,))  =  B(6ob,)  so  that 
b  =  u  =  -$-1(£(0ot»))-  (*(  •)  denotes  the  cumulative  distribution  function  of  .V(0, 1))  Now 
from  (7) 

P{9{0)  ~  9(6)  <  g(t)  -  $(0O»,))  =  $($(*)  -  g(0ob,)  +  b)  (9) 

and  from  (8)  we  obtain 

B(t)  =  P(0*  <t)  =  P(g(0*)  <  g(t))  =  S(ff(t)  -  giL.)  ~  b)  (10) 

Solving  for  g(t)  —  in  (10)  and  substituting  into  (9)  we  have 

P(9(9)  ~  g(6)  <  g(t)  -  g(L.))  =  +  2b)  (11) 

Finally,  to  get  a  1  —  2a  percent  confidence  interval,  we  set  the  right  side  of  (11)  equal  to  a 
and  1  —  a,  and  solve  for  t  to  obtain 

6  e  (B-'mza  -  26)),  B-1($(zi_0  -  26)))  (12) 

where  zp  denotes  the  pth  quantile  of  Interval  (12)  is  called  the  Bias- Corrected  Percentile 
Interval.  The  parametric  assumption  A/(u,  1)  turns  out  to  be  not  as  restrictive  as  it  appears. 
If  we  instead  let  H  =  >/(«,< r2),  with  a 2  unknown,  and  repeat  the  above  derivation,  we  get 
6  =  ~  =  — $-1(£?(0o*#))  and  we  obtain  the  same  interval  (12)  . 

Note  then  when  6  =  0,  the  bias-corrected  percentile  interval  reduces  to  the  percentile 
interval.  Hence  we  can  think  of  the  bias-corrected  interval  as  a  “fine-tuning”  of  the  percentile 
interval. 


5.3.2  Other  Symmetric  Location  Scale  Families . 

In  the  bias-corrected  interval  above,  we  can  just  as  well  assume  that  H  is  some  other 
symmetric,  location  scale  family,  say  H(x  | u,a)  =  This  gives  the  bias-corrected 

interval 

6  e  ( B~l(H0{ha  -  2b)),B~1(Ho(h1-a  -  26)))  (13) 

where  6  =  — fl^'1(5((?0j#))  and  hp  denotes  the  pth  quantile  of  Hq. 

A  natural  question  to  ask  is:  how  much  difference  does  the  choice  of  Hq  make?  Nat¬ 
ural  candidates  to  compare  with  the  normal  are  symmetric,  long  tailed  distributions.  Ben- 
jamini  (1983)  provides  an  appealing  definition  of  long-tailedness.  Suppose  F  and  G  are  both 
symmetric  about  the  origin.  Then  G  is  said  to  stretched  (or  long  tailed)  compared  to  F  if 
^'~1(p)/P~1(p)  *s  311  increasing  function  of  p,  for  1/2  <  p  <  1.  This  definition  reflects  the 
intuitive  meaning  of  long-tailedness,  that  the  quantiles  of  G  are  “farther  out”  than  those  of  F. 
Under  this  definition,  distributions  like  the  f,  logistic  and  cauchy  are  stretched  with  respect  to 
the  normal,  as  we  wduM  expect. 
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Now  suppose  H0  is  stretched  with  respect  to  $.  Assume  B{8ot,)  =  q  >  .5,  so  that  is 

biassed  upward,  and  b  =  —$~1(B(0))  <  0.  Then  the  bias  correction  under  Hq  will  be  in  the 

same  direction  as  the  bias -correction  under  but  will  be  smaller.  The  proof  of  this  fact  is 
easily  derived  from  Benjamini’s  definition  above.  Denoting,  as  before,  the  pth  quantiles  of 
and  H0  by  zp  and  hp  respectively,  we  note  that  H0(ha  +  2 hq)  >  a.  Hence 

*-1(g0(/ta  +  2M)  »~x(«)  _  fu. 

H^(H0(hQ  +  2hq))  Hi'ia)  ha 

This  implies  ^-1(fr0(Ao  +  2 hq))  <  za  +  2 za{hq/ha)  <  za  +  2 zq.  Thus  9(za  +  2 zq)  >  H0{ha  + 
2 hq)  >  a. 

A  similar  argument  shows  that  if  q  <  .5,  then  $(z0  +  2z,)  <  H0{ha  +  2 hq)  <  a,  and  the 
corresponding  results  hold  for  the  upper  quantile.  The  above  proof  requires  that  ha  +  2 hq  <  0. 
This  will  be  the  case  unless  the  bias  in  is  so  large  that  q  is  near  1  —  a. 

The  numbers  in  Table  1  show  the  amount  of  bias  correction  (that  is  {Ho{ha  +  2hq), 
Ho{hi-a  +  2hq))  for  the  normal,  logistic  and  the  cauchy  distributions,  when  a  =  .05. 
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Table  1 


q 

Normal 

Logistic 

Cauchy 

.40 

(.015,  .869) 

(.023,  .884) 

(.045,  .944) 

.45 

(.027,  .916) 

(.034,  .927) 

(.050,  .950) 

.55 

(.084,  .973) 

(.073,  .966) 

(.050,  .950) 

.60 

(.131,  .985) 

(.106,  .977) 

(.056,  .955) 

The  choice  of  a  symmetric  pivotal  distribution  appears  to  make  little  difference.  The 
effect  of  an  assymetric  pivotal  distribution,  however,  can  be  large,  as  Example  1  will  show. 


5.3.3  Another  Justification  for  the  Bias- Corrected  Interval 
In  place  of  A4  and  A5,  we  could  assume 

h(9  -0)~H  (-46) 

and 

h(9*  -  90i,)  ~  H  {A7) 

with  H  symmetric,  and  h  increasing  and  anti-symmetric  {h(~ x)  =  — A(x)).  Letting  H  be  a 
location-scale  family,  we  again  obtain  the  bias-corrected  percentile  interval  (13)  .  When  H  is 
symmetric  around  0,  $  —  $  is  symmetric  around  0  and  the  interval  reduces  to  the  percentile 
interval. 

Finally,  we  could  replace  h(9  -  $)  and  h{9*  -  0O^9)  by  h{9j0)  and  h(9*f0os$)  respectively, 
with  h(l/z)  =  —  h(x)y  and  again  the  bias-corrected  interval  emerges. 
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6.  Comparison  Between  the  Bootstrap  Pivotal  and  Percentile 
Intervals. 

The  bootstrap  pivotal  and  percentile  intervals  differ  in  their  assumptions.  In  constructing 
the  bootstrap  pivotal  interval,  we  had  to  specify  the  exact  form  of  the  pivotal  but  we  assumed 
nothing  about  its  distribution.  On  the  other  hand,  in  building  the  percentile  interval,  knowl¬ 
edge  of  the  exact  form  of  the  pivotal  was  not  necessary  but  we  did  require  that  its  distribution 
be  symmetric  around  0.  For  the  bias-corrected  percentile  interval,  we  weakened  that  assump¬ 
tion  to  one  of  symmetry  around  any  point,  but  we  paid  a  price:  it  was  necessary  to  specify  a 
distribution  for  the  pivotal. 

Which  of  these  intervals  is  better  depends  on  the  problem.  It  is  helpful  to  look  a  few 
simple  examples.  In  each  case,  the  data  are  assumed  to  be  gaussian. 

•  The  Mean:  9  =  E(X),  variance  known.  The  bootstrap  pivotal  interval  based  on  0-0  and 
the  percentile  interval  will  give  very  similar  results,  and  both  will  have  approximately  the 
right  coverage. 

•  The  Correlation  Coefficient :  X  —  (Y,Z)  and  9  =  E(Y  —  E(Y))(Z  —  E(Z))/{E(Y  — 
E(Y))2E(Z  —  E{Z))2}1!2 .  The  random  variable  tanh~l0  —  tanh~l0  is  approximately 
X(9/(2(n  —  3),1  /(n  —  3)).  Hence  the  bootstrap  pivotal  interval  based  on  t(9^9)  = 
tanh~l0  —  tanh~l9  and  the  bias-corrected  percentile  interval  (using  the  normal  family) 
both  should  work  well.  The  uncorrected  percentile  interval  will  be  biassed. 

•  The  Variance:  0  =  E{X— E(X))2 .  The  random  variable  9/0  is  x2_i>  hence  the  bootstrap 
pivotal  based  on  t{9^9)  —  log0  —  log#  will  have  approximately  the  right  coverage.  The 
distribution  logx2  is  not  symmetric,  however,  so  the  percentile  intervals  may  not  work 
well  (see  Example  1).  It  is  clear  that  a  transformation  to  a  symmetric  pivotal  doesn’t 
exist  here  since  such  a  transformation  must  also  remove  the  dependence  of  the  variance 
on  9 .  A  simple  delta  method  calculation  shows  that  only  y(0)  =  log#  achieves  this. 

The  above  examples  represent  some  of  the  problems  that  are  well  understood.  In  most 
situations,  however,  matters  are  much  more  difficult.  To  construct  a  bootstrap  pivotal  interval, 
we  first  need  to  specify  a  quantity  t (X,  9 )  that  is  approximately  pivotal.  This  alone  is  a  difficult 
task  unless  we  know  something  about  the  underyling  distribution.  Now  suppose  we  are  able  to 
specify  a  pivotal  t(X,0 ).  Then  if  t(Xf0)  ~  H  and  t(J^*,^f)  ~  H ,  the  resulting  interval  will 
have  the  correct  coverage.  In  some  problems,  however,  the  bootstrap  distribution  of  t(X%  0oi9) 
can  be  a  poor  approximation  to  H .  One  such  example  is  the  following.  Consider  the  situation 
Xi,X2,...X15  ~  e~l~*  for  x  >  —1.  The  bootstrap  pivotal  interval  for  9  —  E(X)  based  on 
X  —  9  has  poor  coverage  because  the  distribution  of  X*  —  X0i9  is  not  a  good  approximation 
to  the  distribution  of  X  —  9.  This  is  because  the  high  positive  correlation  between  X  and 
the  sample  standard  deviation  S  causes  underestimation  of  the  scale  when  x  is  smaller  than 
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$  and  overestimation  of  the  scale  when  x  is  greater  than  6 .  Basing  the  interval  on  ( X  —  0)jS 
alleviates  this  problem  and  the  resultant  interval  has  good  coverage. 

As  far  as  the  percentile  methods  are  concerned,  an  important  question  is:  when  does  a 
transformation  to  a  pivotal  quantity  exist?  (Efron  (1983)  discusses  normalizing  transforma¬ 
tions).  If  such  a  transformation  existed  for  a  broad  class  of  problems,  the  percentile  (and 
bias-corrected  percentile)  intervals  would  be  attractive  because  they  eliminate  the  problem  of 
having  to  specify  a  pivotal. 

Unfortunately,  we  have  no  solid  answers  to  these  questions;  instead  we  move  on  to  a 
different  technique  that  sidesteps  these  difficulties. 
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7.  Non- Parametric  Tilting  Intervals. 


7.1.  Definition 

As  mentioned  in  the  overview,  the  “ N on-parametric  tilting”  interval  (Efron  (1981))  is 
fundamentally  different  than  the  preceding  methods.  Instead  of  assuming  the  existence  of  a 
pivotal,  it  attempts  to  estimate  the  distribution  of  the  test  function  6  —  $ i  for  each  value  of 
$i.  Then,  as  in  classical  testing  theory,  a  confidence  interval  is  formed  consisting  of  values  of 
$1  for  which  we  would  accept  the  observed  data  in  a  test  of  H  :  8  =  6\. 

We  begin  by  assuming  that  the  true  distribution  F  is  supported  only  on  xi,...xn.  (We’ll 
see  later  why  this  is  necessary).  Let  w  =  (t»i, . ..«>„),  with  10, •  >  0  and  J2iwi  =  1-  Recall 
that  Fn(w)  is  the  distribution  putting  mass  u>,-  on  a:,-,  i  =  1,2 ...n  and  6(w)  =  $(Fn(w)).  Then 
the  true  distribution  F  can  be  represented  by  wfrue,  (i.e.  F  =  Fn(wtrue)).  Let  Pu  denote 
probability  under  Mult(n,  w)/n,  and  denote  the  observed  distribution  by  «°  =  (^,  ...£).  Then 
the  N on-parametric  tilting  interval  is  {Slow  >  Sup)  given  by 

e,ow  =  inf{9x  :  PWti  (*(W)  >  0oi.)  >  a}  (15) 

and 

$„p  =  8up{0 1  :  Pwtl(Hw)  ^  Oobi)  >  «}  (16) 

where  w$1  minimizes  X?(w,«°)  subject  to  ^(«)  =  $i,  and  D(w,  w°)  is  the  (backward)  Kullback- 
Leibler  distance  between  w  and  w°: 

n 

D(w,  w°)  =  Wil°l l(nw*)  (17) 

i 

In  words,  $iow  is  the  smallest  parameter  value  $i  such  that  if  the  data  came  from  a  distribution 
with  parameter  $i,  there  is  probability  at  least  a  of  observing  60t,.  Similarly  for  $„p.  Given 
any  value  $lt  there  may  be  more  than  one  distribution  w  with  6{w)  =  $i,  so  we  use  our  “best 
guess” —  the  closest  distribution  in  Kullback-Leibler  distance  to  our  data,  subject  to  6{w)  =  $i. 

IVom  the  definition  of  the  non-parametric  tilting  interval,  it  certainly  isn’t  clear  why  it 
should  have  the  correct  coverage.  We  investigate  this  in  Section  7.5.  We  will  first  discuss  how 
(0/otri  9vp)  can  be  computed. 
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7.2.  Finding  (0/otP,  0Up) 


In  order  to  compute  (#/ow,  lup)>  we  need  to  find  the  distribution  w  that  minimizes  D(w, «°) 
subject  to  6(w)  =  Oi.  Minimization  by  the  method  of  lagrange  multipliers  yields  weights  of 
the  form 


w 


t 

i 


(18) 


where  Di  =  evaluated  at  w*.  Note  that  the  observed  distribution  w°  corresponds  to 

t  =  0.  The  one  dimensional  family  (18)  contains  the  closest  distributions  to  w°  for  each  fixed 
value  of  0(w).  It  is  illustrated  in  Figure  1. 


Figure  1 


(1,0,0) 


(0,1,0) 


The  triangle  represents  the  simplex  Sn  =  {w  ■  10, •  >  0,  10, •  =  1}  for  n  —  3.  The  statistic 

$  =  6(F„(v))  can  be  thought  of  a  surface  defined  over  the  simplex,  and  the  curve  passing 
through  the  sample  point  «?°  is  the  one-dimensional  family  defined  by  (18)  .  Consider  any 
distribution  (say  w1')  in  this  family.  Suppose  6{wf )  =  ff .  Then  wl  is  the  closest  distribution 
to  ®°  (i.e.  minimizes  D(w,w0))  having  $  value  6*. 


IS 


To  find  6i0„  and  $up,  it  is  necessary  to  compute  Put(6(W)  >  0ojf)  and  Pwt{6(W)  <  6^,) 
for  many  values  of  t.  When  6(W)  is  the  sample  mean,  the  distribution  of  6(W)  under  any  w* 
turns  out  to  be  a  simple  exponential  tilt  of  the  distribution  under  v°.  As  a  result,  it  is  only 
necessary  to  simulate  the  distribution  of  9(W)  under  «°;  the  remaining  distributions  can  then 
be  derived  analytically.  This  is  the  reason  for  the  name  “non-parametric  tilting  interval”  (see 
Efron  (1981)  for  details). 

For  non-linear  statistics,  the  tilting  property  no  longer  holds.  Hence  to  determine  0*^, 
we  need  to  search  along  w *  until  Z\p«(0(W)  ^  $<*$)  is  exactly  equal  to  a.  (and  similarly 
for  $up).  (We  have  labelled  these  distributions  w*lew  and  w*up  in  Figure  1).  Now  for  each  t, 
direct  computation  of  Pwt{$(W)  >  6 oh,)  requires  a  monte  carlo  approximation,  and  this  would 
make  the  search  procedure  far  too  costly.  Instead,  we  can  generate  B  samples  from  a  fixed 
distribution  Vjixtd,  and  use  as  our  monte  carlo  estimate: 

pV'(W)  >  o*,)  =  ^  £  mw{)  >  (is) 

where  W*  are  the  proportion  vectors  of  each  of  the  generated  samples  and  Mw  (W )  indicates 
the  multinomial(n,w)  probability  of  vector  W .  This  estimate  is  unbiassed  since  its  expected 
value  is  Ev^mW*)  >  )}  equals  Pwt(6(W)  >  60i,).  It  reduces 

to  the  usual  Monte  Carlo  estimate  of  Pwt($(W)  >  6ai,)  when  »/,*««/  =  »*• 

Wise  choice  of  can  make  the  variance  of  the  estimate  in  (19)  small.  The  theory  of 

importance  sampling  suggests  that  we  should  try  to  make  the  summand  in  (19)_as  constant  as 
possible.  The  choice  of  wjixej  —  w°  accomplishes  this.  Using  w°  has  the  additional  advantage 
that  if  a  percentile  interval  has  been  found  for  a  given  data  set,  no  further  function  evaluations 
are  needed  to  find  the  tilting  interval. 


7.3.  Approximating  the  derivatives  D,- 

In  the  above,  we  have  ignored  the  fact  that  D,-  is  a  function  of  w,  so  that  (18)  defines 
w *  only  implicitly.  As  an  approximation  to  Z),-,  we  use  Z?®  =  evaluated  at  w°.  If  9{w) 
is  too  complex  to  make  calculation  of  D ®  possible,  we  can  use  the  jackknife  estimate  of  the 
directional  derivative  Uf  given  by 

Z7,  =  (n-l)[te(w_,))-^.]  (20) 

where  «>_,•  =  (l/(n  -  1),  1  /{n  —  1),  ...l/(r»  -  1),  0,  l/(n  -  1),  ...l/(n  -  1)),  zero  being  in  the  ith 
place.  Since  D °  =  U,+constant,  (see  Efron(1982)  pg  37-41  for  details),  t/,-  can  be  used  in  place 
of  Z?°  in  formula  (18)  . 
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7.4.  The  Non-parametric  Tilting  Interval  as  a  Generalization  of  the  Boot¬ 
strap  Pivotal  Interval 

The  interval  (0io„,0 up)  can  be  derived  as  a  generalization  of  a  bootstrap  pivotal  interval 
based  on  6  —  8.  In  the  bootstrap  pivotal  case,  we  proceeded  as  follows.  We  began  by  assuming 
that  0  —  0  ~  H,  H  not  involving  6.  We  then  estimated  H  by  assuming  that  the  bootstrap 
version  of  this,  6*  -  0O jf  also  had  the  distribution  H.  Using  this  bootstrap  distribution,  the 
pivotal  was  inverted  to  yield  the  interval.  Note  that  the  bootstrap  approximation  consisted 
solely  of  estimating  the  unknown  F  by  F„,  the  unrestricted  maximum  likelihood  estimator  of 
F. 

In  the  present  case,  we  don’t  assume  that  6{W)  —  8  is  pivotal,  and  hence  we‘ll  write 
0(W)  —  0  ~  H$  to  indicate  that  H  depends  on  0.  In  order  to  construct  the  interval,  we  need 
to  estimate  Hgx  for  each  trial  value  0\.  To  do  so,  we  first  compute  the  restricted  maximum 
likelihood  estimator  of  F,  subject  to  6(F)  =  0\.  Since  we  are  assuming  that  F  has  support  only 
on  xi,  x2,  —xn,  this  restricted  m.l.e.  corresponds  to  a  set  of  weights,  say  wgl.  Proceeding  as  in 
the  pivotal  case,  we  assume  that  under  w$l ,  0(W )  -  also  has  distribution  Hgl .  Now  to  obtain 
the  confidence  interval,  we  can’t  just  “invert  the  pivot”  since  H$t  depends  on  0\.  Instead, 
we  consider  each  8 1  separately,  including  0i  in  the  confidence  interval  if  PwSl  (8(W )  —  6%  > 
8obt  ~  8i)  >  a  and  Pv>t  (8(W)  —  8 x  <  6 ot,,  —  0i)  >  a-  Finally,  equivalence  between  maximizing 
the  multinomial  likelihood  and  minimizing  the  Kullback-Leibler  distance  almost 

yields  (15)  and  (16)  .  We  say  “almost”  because  maximizing  L  subject  to  0(w)  =  $i  is 
equivalent  to  minimizing  the  “forward”  Kullback-Leibler  distance  D(w°,w)  =  Z)  £l0g 
subject  to  6(w)  =  Strictly  for  computational  reasons  *  we  instead  use  the  “backward” 
Kullback-Leibler  distance  Z?(w,id0). 

Note  that  if  we  don’t  restrict  the  support  of  F  to  xi,X2,...Xn)  the  maximum  likelihood 
estimator  of  F  subject  to  6(F)  =  $i  is  pathological  in  many  cases,  and  not  of  much  use  in 
constructing  the  interval.  Alternatively,  we  can  impose  smoothness  restrictions  on  F  (see 
Section  7.6). 

Figure  2  illustrates  the  level  surfaces  G$  of  constant  $  (C$  —  {»  :  $(*>)  =  $})  together  with 
the  various  quantities  used  in  deciding  whether  to  include  6  in  ( 6i0w,6up ).  (The  distribution 
appearing  in  Figure  2  should  be  ignored  for  now —  it  will  be  important  in  discussing 
a  variant  of  this  interval  in  Section  7.6).  Figure  2  makes  clearer  the  distinction  between 
the  bootstrap  pivotal  and  non-parametric  tilting  intervals.  Both  intervals  use  the  distance 
function  6(W)  -6  to  measure  the  distance  between  a  point  W  and  the  level  surface  C$.  The 
bootstrap  pivotal  method  assumes  that  0(WK)  -  6  has  a  distribution  not  depending  on  $ ,  i.e. 
the  distribution  of  6(W)  -  6  is  the  same  no  matter  which  level  surface  Wtrue  is  on.  As  a  result, 
we  don’t  have  to  estimate  the  distribution  of  W  for  each  0,  and  we  need  only  sample  from  the 


*  The  forward  Kullback-Leibler  distance  yields  weights  of  the  form  cf(l+tDi)  which  we  have  found  difficult 
to  deal  with  computationally. 
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distribution  w°  to  deduce  the  interval.  On  the  other  hand,  the  non-par ametric  tilting  interval 
doesn’t  assume  6{W )  —  0  is  pivotal,  hence  we  have  to  1)  estimate  the  distribution  of  W  for 
each  6—  our  estimate  is  wg  and  2)  sample  from  each  wg  to  determine  whether  to  include  6  in 
the  confidence  interval. 


Figure  2 


7.5.  Coverage  of  tlie  Non-Parametric  Tilting  Interval 

Will  the  interval  {0ioa,  0up)  cover  the  true  value  with  probability  at  least  1  -  2a,  for  all 
Wtrue?  To  investigate  this,  we  must  state  clearly  what  we  mean  by  a  new  realization  of  the 
interval.  The  points  xlf  x2, .  •  •  x„  are  fixed,  and  the  true  distribution  wlrue,  has  mass  only  on 
xl,z2...x„.  A  new  sample  from  wtrue  corresponds  to  some  weight  vector  wnea.  From  this 
new  sample  there  will  correspond  Wgew,  the  closest  distribution  to  wnew  on  the  level  surface 
Cg  for  some  fixed  6.  Now  let  t(6)  be  the  upper  a  percent  point  of  the  distribution  of  $(W) 
under  wnew.  Then  the  non-parametric  tilting  will  not  include  $  on  the  left  (i.e.  6 tow  will 
be  greater  than  $)  if  0oi,  >  t{6).  The  miscoverage  probability  on  the  left  will  therefore  be 
/  P(0{W)  >  i($))dQ(t(6)  where  Q(t(0))  is  the  distribution  of  t(0)  tinder  wt,Uc  (And  similarly 
for  the  miscoverage  on  the  right.)  This  quantity  can  not  be  computed  analytically,  but  we 
can  see  that  two  factors  will  determine  the  coverage:  1)  how  far  away  the  vgew's  are  from 
w,rae  and  2)  how  much  the  t(0)’s  change  away  from  wlrut.  (Note  that  if  each  v™*  equalled 
Wtrue,  t{6)  would  equal  the  upper  percent  point  of  0{W)  under  «w,  and  the  coverage  would 
exactly  equal  a.)  A  better  understanding  of  these  factors  might  lead  to  improvements  in  the 
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non-parametric  tilting  interval.  One  such  attempt  is  made  in  Section  7.7. 


7.6.  Smooth  Tilted  Intervals 

Instead  of  assuming  that  F  is  supported  on  Xi,i2,  ...x„,  we  could  assume  instead  that 
F  belongs  to  a  smooth  family.  A  convenient  choice  for  this  family  is  F„(w)  convolved  with  a 
symmetric  kernel  k(-): 

»•<»>-??*  (4*)  («> 

The  window  parameter  h  can  be  estimated  from  the  data  or  more  simply,  a  reasonable  fixed 
value  (say  h  =  .25)  can  be  used. 

The  form  of  the  weights  that  minimize  D{w, «°)  =  Egw  log  gw/dte0  appears  quite  difficult 
to  obtain  for  general  kernels  k.  (Since  the  weights  are  to  be  used  as  part  of  a  computationally 
intensive  procedure,  an  explicit  expression  is  required.)  All  is  not  lost,  however.  When  the 
supports  of  the  k[vjfiys  are  disjoint,  the  minimizing  weights  are  exactly  of  the  form  (18)  , 
independent  of  Jfc(-).  Thus,  as  an  approximation,  we  can  use  weights  of  the  form  (18)  .  This 
should  be  adequate  for  small  h. 


7.7.  Another  Variant  of  the  Non-Parametric  Tilting  Interval 

A  variant  of  the  {$iow,  8up)  interval  can  be  obtained  by  use  of  a  test  function  other  than 
6(W)  —  8.  Recall  that  w$  is  the  closest  point  (in  backward  Kullback-Leibler  distance)  on  C$ 
to  *o°.  Then  a  natural  measure  of  the  plausibility  of  a  value  8  is  the  distance  D(w$,w°).  In 
fact,  we  will  also  want  to  take  into  account  on  which  “side”  of  C$  the  observed  distribution 
®°  lies.  This  can  be  achieved  by  attaching  a  sign  to  the  distance  in  some  consistent  fashion. 
The  interval  defined  below  is  similar  to  the  one  introduced  by  Efron  (1984)  for  a  certain  class 
of  parametric  problems. 

For  an  arbitrary  distribution  w,  let  w$tW  be  the  closest  point  in  backward  Kullback-Leibler 
distance  on  C8  to  v.  Define  the  signed  (backward)  Kullback-Leibler  distance  by  SD(w$tW,w)  = 
D(v0tW,w)  if  8  >  8(w)  and  -D(wgtV},w)  otherwise.  Now  consider  a  trial  value  61.  Then  we 
include  $i  in  our  confidence  interval  if 

PWtiW(SD{w$w,W)  >  SD(w9l,w0))  >  a  (22) 

and 

Pwt  W(SD(w8yW,W)  <  SD(w8l,w0))  >  a  (23) 

Figure  2  again  illustrates  this,  the  only  difference  being  that  the  distance  from  a  point  W  to 
the  level  surface  C8  is  measured  by  minimum  Kullback-Leibler  distance  instead  of  by  $ (W )  -  8 . 
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How  would  this  interval  compare  to  the  (0iow  >  0«p)  interval?  If  the  level  curves  are  such 
that  SD(wg  w>w)  =  0  —  (or  some  monotone  function  of  8  —  0(W))  then  the  two  methods 

are  identical.  This  follows  simply  by  the  definition  of  the  two  intervals.  In  general,  however, 
they  will  be  different.  The  parametric  analogues  of  these  procedures  suggest  that  the  interval 
based  on  SD(wejp,W)  might  be  less  sensitive  to  departures  of  w$  from  Wtrae  and  to  the 
changing  shape  of  the  level  surfaces.  On  the  other  hand,  it  appears  to  be  much  more  difficult 
to  compute.  The  reason  is  that  for  each  bootstrap  sample,  the  distance  SD(w8jp ,W)  must 
be  computed,  and  this  requires  an  expensive  search.  (Only  one  such  search  was  necessary 
in  computing  the  non-parametric  tilting  interval.)  Some  clever  computing  tricks  might  make 
computation  of  this  interval  possible,  but  this  is  still  under  investigation. 


8.  Estimated  Pivotal  Intervals. 


8.1.  Definition 

We  propose  here  a  compromise  between  the  pivotal  methods  and  the  non-parametric  tilt¬ 
ing  technique.  The  idea  is  to  use  the  estimated  distributions  provided  by  the  tilting  technique 
to  estimate  the  form  of  a  pivotal.  This  pivotal  in  then  used  as  the  basis  for  a  bootstrap  pivotal 
interval  (as  described  in  Section  4). 

Assume  6  —  6  H  and  6*  -  ~  H  where  H  has  variance  v(6).  Then  a  one  term  Taylor 

expansion  shows  that  g(0)  —  g[6)  has  approximately  constant  variance,  where 

m  =  ['  -K  *  <m) 

Je  v(t)» 

A  convenient  family  for  v(6)  is  vp(6)  =  \0  |p  for  0  <  p  <  2.  Then  gp(0)  =  \ 6  \ 1_p/2  if 
p  ^  2  and  g2[0)  =  log  \0  |. 


8.2.  Estimation  of  p 

We  use  the  family  of  closest  distributions  to  w°  to  estimate  v(6)  for  various  values  of 
$.  These  estimates  are  obtained  using  the  computational  trick  of  Section  7.  A  simple  linear 
regression  of  log  t>(0)  on  \6  |  then  provides  an  estimate  of  p. 

Given  the  estimate  p,  we  then  utilize  the  pivotal  gp{6)  -  gp(6)  in  the  bootstrap  pivotal 
interval  given  by  (2)  . 
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Section  9:  Some  Numerical  Examples 


9.  Some  Numerical  Examples. 

To  illustrate  the  various  methods  discussed  here,  we  compared  them  in  two  problems. 


Example  1.  The  Variance :  6  =  E{X  -  E(X))2,  X  ~  JV(0, 1). 

As  we  discussed  earlier,  we  would  expect  that  the  bootstrap  pivotal  interval  based  on 
6/6  to  perform  well,  since  6/0  is  a  pivotal,  but  would  expect  the  percentile  and  bias-corrected 
percentile  intervals  not  to  do  as  well. 

We  performed  a  small  simulation  study  to  investigate  this.  1000  samples  of  size  20  were 
generated  and  central  90  percent  confidence  intervals  were  constructed  for  each  sample.  The 
results  are  shown  in  Table  2.  The  normal  theory  interval  obtained  by  pivoting  6/0  around  the 
percentage  points  of  Xio  can  be  thought  of  as  the  “correct”  interval  and  had  actual  coverage 
close  to  90  percent.  The  bootstrap  pivotal  interval  does  fairly  well,  while  the  others  display 
too  low  coverage.  The  percentile  interval  is  especially  poor.  The  non-parametric  tilting  and 
estimated  pivotal  intervals  captured  the  asymmetry  of  the  normal  interval  better  than  the 
bias-corrected  interval,  but  their  coverage  was  still  too  low. 

It’s  not  surprising  that  the  intervals  based  on  6  —  0  or  g(6)  —  g(0)  didn’t  perform  well: 
this  particular  problem  is  difficult  because  the  variance  of  0  depends  on  0 .  Only  the  bootstrap 
pivotal  interval,  which  utilizes  knowledge  of  this  fact,  performs  satisfactorily. 

Table  2  also  displays  the  results  for  the  “smooth”  versions  of  these  intervals.  All  boot¬ 
strapping  was  done  from  Fn(w)  *  .25J/(O,0o*f)  instead  of  Fn(»).  For  the  bootstrap  pivotal, 
percentile  and  bias-corrected  percentile  intervals,  this  meant  sampling  a  data  value  with  re¬ 
placement  from  X\ ,  a: 2 >  then  adding  .25  times  a  M{0,6of,$)  random  variable.  The  approxi¬ 

mate  weights  (18)  were  used  for  the  non-parametric  tilting  interval.  The  smoothing  improved 
the  coverage  of  all  the  bootstrap  intervals,  but  surprisingly,  it  pulled  the  bias-corrected  and 
non-parametric  tilting  intervals  away  from  the  normal  interval. 

Finally,  we  tried  the  unsmoothed  intervals  with  sample  50.  All  the  intervals  performed 
quite  well  except  the  percentile  interval  which  was  biassed. 

This  problem  was  also  studied  by  Schenker(1983).  He  obtained  results  for  bootstrap 
pivotal  and  percentile  intervals;  they  are  in  qualitative  agreement  with  those  given  here. 
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Example  2.  The  10  percent  Trimmed  Mean 

1000  Samples  of  size  20  were  generated  from  the  contaminated  normal  .901/(0,1)  + 
.101/(0,3)  and  central  90  percent  confidence  intervals  were  computed.  For  comparison  with 

A  A 

the  bootstrap  interval,  we  computed  the  asymptotic  interval  (0  -  t.85*yt,  0  +  t.95 «/*)  where 
t.95  is  the  95th  percentile  of  the  tX9  distribution  and  is  the  jackknife  estimate  of  standard 
deviation.  The  results  are  shown  in  Table  3.  All  the  intervals  had  approximately  the  right 
coverage;  surprisingly,  the  bias-correction  seemed  to  make  the  percentile  interval  worse. 


10.  Closing  Remarks. 

We  have  discussed  a  number  of  bootstrap  techniques  for  constructing  confidence  intervals. 
All  are  potentially  useful  as  data-analytic  tools  because  they  are  non-parametric  and  can  be 
applied  in  complex  situations.  Further  work  is  needed  to  evaluate  and  improve  these  methods. 
Our  current  research  focusses  the  non-parametric  tilting  interval  and  its  variants. 
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Table  2 


Confidence  Intervals  for  the  Sample  Variance 

Sample  Size:  20 

Monte  Carlo  size:  1000 

Number  of  Bootstraps:  500 

Ave  Lover  Ave  Upper  Level  (Standard  Error) 


Normal 

.632 

1.883 

9.8% 

(0.94) 

Pivotal 

.670 

1.860 

15.7% 

(1.15) 

Percentile 

.496 

1.357 

23.3% 

(1.34) 

Bias-C  Percentile 

.595 

1.490 

20.9% 

(1.29) 

Tilting 

.634 

1.552 

20.2% 

(1.27) 

Estimated  Pivotal 

.587 

1.520 

19.3% 

(1.25) 

Pivotal  Smoothed 

.625 

1.752 

13.8% 

(1.09) 

Perc  Smoothed 

.525 

1.452 

19.3% 

(1.25) 

Bias-C  Smoothed 

.539 

1.472 

18.8% 

(1.24) 

Tilting  Smoothed 

.568 

1.527 

17.4% 

(1.20) 

Sample  Size:  50 


Normal 

.723 

1.404 

10.3% 

(0.96) 

Bootstrap  Pivotal 

.750 

1.431 

11.7% 

(1.02) 

Percentile 

.670 

1.272 

16.9% 

(1.19) 

Bias-C  Percentile 

.718 

1.335 

14.7% 

(1.12) 

Tilting 

.738 

1.372 

13.4% 

(1.08) 

Estimated  Pivotal 

.718 

1.341 

13.3% 

(1.07) 

Table  3 

Confidence  Intervals  for  the  10%  tr limned  mean 

Sample  size:  20 

Monte  Carlo  Size:  1000 

Number  of  Bootstraps:  500 


Ave  Lower  Ave  Upper  Level  (Standard  Error) 

.4590  10.1%  (0.95) 

.4764  8.3%  (0.87) 

.4777  10.7%  (0.98) 

.4953  8.3%  (0.87) 

.4694  11.1%  (0.99) 


T- Interval  -.4427 
Bootstrap  Pivotal  -.4585 
Percentile  -.4616 
Blas-C  Percentile  -.4825 
Tilting  -.4510 


